fix: bound worker hello and cancel reconnect backoff - #353
Merged
wangbill (YunchuWang) merged 18 commits intoAug 27, 2026
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR changes TaskHubGrpcWorker.start() semantics so that awaiting startup only resolves after the sidecar “hello” handshake succeeds and the initial work-item stream is confirmed ready, with a configurable startup timeout budget (default 30s). It also wires the new timeout knob through the Azure-managed worker builder and updates docs/changelogs accordingly.
Changes:
- Make
await worker.start()wait for hello + initial work-item stream readiness, with a boundedstartupTimeoutMscovering metadata generation, hello, and stream readiness. - Add comprehensive unit tests for startup timing, cancellation, late callbacks, and stream recovery behavior.
- Expose
.startupTimeout(milliseconds)onDurableTaskAzureManagedWorkerBuilderand document the new startup semantics.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| README.md | Documents new start() readiness semantics and builder timeout configuration. |
| CHANGELOG.md | Notes the core worker startup behavior change. |
| packages/durabletask-js/src/worker/task-hub-grpc-worker.ts | Implements startup readiness gating, startup timeout budget, and generation-based cancellation/cleanup. |
| packages/durabletask-js/test/worker-startup.spec.ts | Adds focused tests covering startup readiness, timeouts, stop-during-start, and late callbacks. |
| packages/durabletask-js/test/worker-stream-recovery.spec.ts | Adjusts mocks to account for metadata-based “stream ready” detection. |
| packages/durabletask-js-azuremanaged/src/worker-builder.ts | Adds .startupTimeout() builder method and passes it into core worker options. |
| packages/durabletask-js-azuremanaged/test/unit/worker-builder.spec.ts | Verifies builder propagates startup timeout to the core worker instance. |
| packages/durabletask-js-azuremanaged/README.md | Documents startup readiness and builder timeout configuration. |
| packages/durabletask-js-azuremanaged/CHANGELOG.md | Notes the new builder API for startup timeout configuration. |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.
Suppressed comments (2)
packages/durabletask-js/src/worker/task-hub-grpc-worker.ts:425
- The start()
@throwsdoc is too narrow: start() can now throw TimeoutError for metadata generation and work-item stream establishment timeouts as well, not only the hello handshake.
* @throws {@link TimeoutError} if the hello handshake exceeds `startupTimeoutMs`.
packages/durabletask-js/src/worker/task-hub-grpc-worker.ts:71
- The TaskHubGrpcWorkerOptions.startupTimeoutMs doc comment no longer matches behavior: the timeout budget is also applied to work-item stream metadata generation and establishing the initial stream, not just hello/metadata. Updating the comment helps avoid confusing API consumers.
/**
* Optional timeout in milliseconds for initial metadata generation and the sidecar hello handshake.
* Defaults to 30000.
*/
startupTimeoutMs?: number;
Make worker startup await the initial sidecar stream, bound connection setup with a configurable timeout, and cleanly abort failed or stopped attempts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Treat an idle stream as ready once hello succeeds and handlers are attached, preserve retryability after client construction failures, and align the recovery E2E with the startup contract. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 65b88a3f-d79a-4a63-8636-cc41c219db63
Add an isolated job that leaves the emulator stopped initially so the worker startup failure and subsequent stream recovery path execute in CI. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 65b88a3f-d79a-4a63-8636-cc41c219db63
Document the startup rejection as a breaking change, align timeout API wording, and preserve the orchestration reuse documentation while rebasing onto main. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 65b88a3f-d79a-4a63-8636-cc41c219db63
wangbill (YunchuWang)
force-pushed
the
yunchuwang-fix-worker-startup-timeout
branch
from
August 10, 2026 23:02
bbef765 to
3482ecb
Compare
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d491ec0d-9ce9-421b-9953-7d179d53625b Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
wangbill (YunchuWang)
force-pushed
the
yunchuwang-fix-worker-startup-timeout
branch
from
August 11, 2026 00:17
45003c4 to
82e60fd
Compare
Track direct internal worker runs under the same lifecycle invariants as start and reconnect so stop can drain them and later starts cannot race stale streams. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d491ec0d-9ce9-421b-9953-7d179d53625b Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d491ec0d-9ce9-421b-9953-7d179d53625b
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d491ec0d-9ce9-421b-9953-7d179d53625b
wangbill (YunchuWang)
changed the title
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d491ec0d-9ce9-421b-9953-7d179d53625b
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d491ec0d-9ce9-421b-9953-7d179d53625b
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d491ec0d-9ce9-421b-9953-7d179d53625b
Restores the existing event-driven worker structure while keeping per-Hello deadline and stop cancellation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d491ec0d-9ce9-421b-9953-7d179d53625b
Restores the existing event-driven worker structure while keeping per-Hello deadline and stop cancellation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d491ec0d-9ce9-421b-9953-7d179d53625b
wangbill (YunchuWang)
changed the title
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d491ec0d-9ce9-421b-9953-7d179d53625b
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d491ec0d-9ce9-421b-9953-7d179d53625b
Keep the worker startup path to one helper call and rely on native Promise settlement for cancellation races. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d491ec0d-9ce9-421b-9953-7d179d53625b
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d491ec0d-9ce9-421b-9953-7d179d53625b
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d491ec0d-9ce9-421b-9953-7d179d53625b
kaibocai (kaibocai)
approved these changes
Aug 27, 2026
wangbill (YunchuWang)
deleted the
yunchuwang-fix-worker-startup-timeout
branch
August 27, 2026 20:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
stop()aborts the current runReview guide
backoff.util.ts: add optionalAbortSignalsupport to the existing reconnect delay.grpc-helper.util.ts: add optional signal cancellation to the existing unary-call Promise helper.task-hub-grpc-worker.ts: create one signal per start and make Hello a single helper call that supplies the 30-second deadline.The final worker diff is 58 additions / 24 deletions; the helper diff is 22 additions / 8 deletions. The inline Hello cancellation state machine,
_helloCall,WorkerLifecycle, connection-loop rewrite, and drained-stop contract are absent.Scope
This intentionally preserves the existing non-blocking
start()contract and event-driven worker structure. It does not add startup readiness semantics or a drained connection-loop lifecycle, so it is related to but does not close #69.Validation